1. Field of the Invention
This invention relates generally to the field of computer graphics and, more particularly, to high performance graphics systems which generate images for display via one or more display devices, including projection devices.
2. Description of the Related Art
Multiple projection devices may be used to display a single integrated image as shown in FIG. 1. Each of the projection devices may display a portion of the integrated image. For example, projection devices P1 and P2 may target a common display screen SCR, and may generate the left and right portions respectively of an integrated image. The projection devices may use any of a variety of technologies. For example, the projection devices may be based on cathode ray tubes (CRTs), liquid crystal displays (LCDs), digital micro-mirror displays, or liquid crystal on silicon (LCOS) displays, etc.
A computer system CS may be used to drive the multiple projection devices P1 and P2. The computer system CS may provide analog or digital video signals to drive the projection devices P1 and P2. The computer system CS may include a graphics system for the rendering and display of 2D graphics and/or 3D graphics. The graphics system may supply the video signals which drive the projection devices P1 and P2. In addition, the computer system CS may include a monitor device MD, a system unit SU, and input devices such as a keyboard KB and mouse MS. Monitor device MD and projection devices P1 and P2 may be referred to generically as display devices.
It is often necessary to overlap the images generated by the two projection devices P1 and P2 as shown in FIG. 2A. A front view of the two overlapped images as they might appear on the display screen SCR is shown in FIG. 2B. The left image represented by rectangle ABCD overlaps the right image represented by rectangle EFGH.
FIG. 2C illustrates a fundamental problem associated with the overlapping of two projected images. The function It represents the distribution of light intensity across the screen SCR due to projection device P1. Similarly, function 12 represents the distribution of light intensity across the screen SCR due to projection device P2. In the overlap region (i.e. rectangle EBCH) the intensity functions add. Thus, the intensity in the overlap region may be larger in the non-overlap regions than in the non-overlapped regions of the screen. The increased brightness of the overlap region may be annoying to a viewer of the integrated image AFGD.
The overlap-brightness problem exists when any number of projected images are overlapped. For example, FIG. 3 illustrates the overlapping of four projected images generated by four projection devices in a “two by two” configuration. One of the projected images is represented by rectangle JLTR. The other three projected images have the same size as rectangle JLTR and are displaced to the three remaining corners M, V and Y. Exactly two of the projected images overlap in regions KLPO, NOSR, STXW and PQUT, and thus, these regions may exhibit a light intensity which is double that of the non-overlap regions JKON, LMQP, RSWV and TUYX. All four projected images overlap in the center region OPST, and thus, the center region may exhibit a light intensity which is four times larger than the intensity of the non-overlap regions.
There currently exist analog systems called “video boxes” which address in some measure the overlap-brightness problem. A video box may couple between computer system CS and projection devices P1 and P2 as shown in FIG. 1, and may reduce the intensity of the projected images in the overlap region. Thus, the total intensity in the overlap region may be more nearly equal to the intensity of the non-overlap regions after installation of a video box. Although, video boxes provide some measure of correction to the overlap-brightness problem, they may be expensive and may not be easy to configure. Furthermore, their functionality may be limited. Thus, there exists a need for a system and method which could provide an improved blending of the light intensity in the overlap regions. More generally, there exists a need for a system and method which could uniformize the intensity distribution of an integrated image generated by multiple projection devices.
In FIG. 2C, the intensity distributions I1 and I2 are illustrated as having rectangular profiles. In reality, the intensity distribution of a projected image may be highly non-uniform for a number of different reasons. First, a typical projection device PD radiates light with a non-uniform radiation intensity pattern as shown in FIG. 4A. The radiation intensity pattern may have a maximum in the on-axis direction, i.e. the direction of central projection ray CPR, and may decrease as a function of the radiation angle RA measured with respect to the central projection ray CPR.
Thus, as shown in FIG. 4B, a more realistic screen intensity distribution for a projected image may have a maximum at the point CM where the central projection ray intercepts the screen SCR, and may decrease as a function of horizontal and vertical screen displacements from the central maximum CM. Several level curves of the screen intensity distribution are superimposed on the projected image. FIG. 4C illustrates the screen intensity distribution ID along a horizontal slice HS of the screen through the central maximum CM.
If the screen material is highly diffusive, the screen intensity distribution ID may be relatively independent of the viewing angle and/or position. Diffusivity means that a generic ray R impinging on the screen is scattered with a hemispherical scattering distribution SD. However, if the screen material has high gain, i.e. is more reflective (or transmissive in the rear-projection case), the screen intensity distribution may be highly dependent on the viewer's position with respect to the projector-screen system.
For example, FIG. 5A illustrates the screen intensity distributions IV1 and IV2 perceived by viewers V1 and V2 respectively situated on opposite sides of projection device PD in a front-projection scenario. The screen intensity distribution IV1 for the first viewer V1 may attain a maximum value at the point P1 where the source ray S1 emanating from the projection device has a reflection ray R1 which intercepts the viewer V1. Thus, the viewer V1 perceives a “hot spot” on the screen at point P1. Similarly, the screen intensity distribution IV2 for the second viewer V2 may attain a maximum value at the point P2 where the source ray S2 emanating from the projection device has a reflection ray R2 which intercepts the viewer V2. Thus, viewer V2 perceives a hot spot at point P2 on the screen SCR.
FIG. 5B illustrates the screen intensity distributions IV1 and IV2 perceived by viewers V1 and V2 respectively situated at different positions with respect to a high gain screen in a rear-projection scenario. The screen intensity distribution IV1 for the first viewer V1 may attain a maximum value at the point Q1 where the direct ray D1 from the projection device to the viewer V1 intercepts the screen. Similarly, the screen intensity distribution IV2 for the second viewer V2 may attain a maximum value at the point Q2 where the direct ray D2 from the projection device to the viewer V2 intercepts the screen.
Multiple projector devices may be used to generate an integrated image on the projection screen SCR. Thus, an observer may observe two or more hot spots on the screen, i.e. the intensity distribution may have two or more local maxima. Thus, there exists a need for a system and method which could correct the hot spots perceived by the observer on screen SCR.
Projection devices generally attempt to project a rectangular array of physical pixels on the projection screen SCR. However, many distortion mechanisms conspire to defeat this desired end. For example, the physical pixel array may exhibit a keystone distortion KST, a pincushion distortion PNC or a barrel distortion BRL as suggested by FIG. 6A. Fundamental distortion modes may combine to form more complex distortions as suggested by combined distortion CDST.
A keystone distortion is synonymously referred to herein as a trapezoidal distortion. As used herein the term “keystone distortion” is intended to include any distortion where the nominally rectangular pixel array is deformed into a trapezoid. The parallel edges of the trapezoid may be either the top and bottom edges, or, the right and left edges. The terms “pincushion distortion” and “barrel distortion” are terms which are familiar from fields such as optics and video projection. Thus, the exact definition of these terms may depend on the context in which they are used. As used herein, the term “pincushion distortion” is intended to include any distortion where the right and left edges, and/or, the top and bottom edges, of the nominally rectangular pixel array experience an inward bending as presented on screen SCR. For example, a CRT display distortion where the right and left sides are bent inwardly while horizontal lines remain relatively straight is referred to herein as a pincushion distortion. As another example, a lens-induced distortion where all four edges are bent inwardly is also referred to herein as a pincushion distortion. As used herein, the term “barrel distortion” is intended to include any distortion where the right and left edges, and/or, the top and bottom edges, of the nominally rectangular pixel array experience an outward bending. For example, a CRT distortion where the right and left sides are bent outwardly, and a lens-induced distortion where all four sides are bent outwardly, are both referred to herein as barrel distortions.
Distortions of the displayed pixel array may arise from a variety of physical mechanisms and have a large variation in magnitude. For example, in a scenario where a projection device projects onto a screen which is not centered on or perpendicular to the projection axis, the resulting geometric distortion may amount to a significant fraction of the displayed pixel array. Other distortions of the displayed pixel array may be subtle, i.e. small in magnitude. Thus, there exists a need for a graphics system and methodology which could compensate for arbitrary physical pixel distortions so that rendered objects would appear undistorted on the screen SCR.
As noted above, multiple projection devices may be used to generate an integrated image or integrated sequence of video images on screen SCR as shown in FIG. 1. In general, the physical pixel arrays generated by the projection devices are overlapped as suggested by the example of FIG. 6B. The left physical array LPA (denoted by small circles) is generated by a first projection device, and the right physical array RPA (denoted by small triangles) is generated by a second projection device. Each physical array may be distorted in an arbitrary manner. Thus, images displayed in the left and right physical arrays may appear distorted. For example, the left physical array LPA is illustrated with a barrel distortion, and the right physical array is illustrated with a keystone (i.e. trapezoidal) distortion.
Even if the distortion of each physical array is small, there may be relative distortions between the left and right physical arrays. For example, the left and right physical arrays may not have the same vertical and/or horizontal size, or one physical array may be translated with respect to the other. As another example, the distortions of the left and right physical arrays may arise from manufacturing imperfections in the light processing chips of each projection device. Thus, images (or portions of images) appearing in the overlap region may appear blurred (i.e. fuzzy). Thus, there exists a need for a graphics system which could compensate for arbitrary physical distortions (including relative distortions) inherent in multiple projection configurations so that the integrated image generated by the multiple projection devices may appear undistorted and consistent, especially in overlap regions. In other words, there exists a need for a system and method which could correct for the independent physical distortions arising in multiple projector scenarios.
The area covered by the left physical array LPA and right physical array RPA on the screen SCR is the effective viewing area. In order to maximize the viewing area, it would be desirable to minimize the size of the overlap between the left physical array LPA and the right physical array RPA. However, if the overlap region is minimized, it becomes difficult to perform edge blending with the prior art video box technology. Thus, there may appear an uneven line of brightness along the interface of the physical arrays. The brightness is due to a higher density of physical pixels. Thus, there exists a need for a system and method which could neatly match the boundaries of physical images generated by multiple projection devices configured with minimum overlap and without generating a line of bright distortion along the interfaces.
Modern projection devices generate color images by a variety of different mechanisms which lead to different modes of chromatic distortion. As used herein the term “chromatic distortion” is intended to include any distortion which results in a spatial separation of displayed color components of physical pixels on a screen. The following is a discussion of several economically viable projection technologies and their chromatic distortion modes.
According to one model, a projection device may comprise three separate color beams which are projected through three respective lenses (or lensing systems) onto the screen SCR. For example, a CRT-based projector may have separate red, green and blue displays, i.e. three separate guns targeting corresponding phosphor surfaces. Each display generates a source image of the corresponding color which is projected onto the screen SCR through a corresponding lens. The projection of each source image onto screen SCR is referred to herein as a physical image. For example, the red source image from the red display is projected onto a red physical image on screen SCR. Ideally, the red, green and blue physical images on screen SCR would exactly overlay one another, generating a single rectangular array of pixel positions. Corresponding pixels in each of the three physical images would cover the same area on screen SCR. However, real projection devices depart from this ideal behavior because of distortions inherent in the displays, the lenses, the alignment of the displays with respect to the lenses, the alignment of the lenses with respect to the screen, etc. Thus, the red, green and blue physical images are independently distorted on SCR, and corresponding pixels of the three physical images fail to overlap on screen SCR.
According to a second model, a projection device may include a single embedded display (e.g. an integrated LCD display) and an optical subsystem. The embedded display generates a source image comprising an array of source pixels. The optical subsystem projects the source image onto screen SCR. Each source pixel may radiate a combination of red, blue and green light spectra. Because optical materials (such as lenses) in the optical subsystem bend light rays in a wavelength dependent fashion, a single source pixel may be fragmented into separate red, green and blue physical pixels on screen SCR. This effect is known as chromatic aberration. Chromatic aberration is responsible for a blurring of images on screen SCR.
According to a third model, a high performance projector may use dichroic optics as follows. A single source beam of white light may be separated into three subbeams. The three subbeams are independently modulated with color information resulting in a red subbeam, a green subbeam and a blue subbeam. For example, the three white subbeams may be transmitted through three corresponding ILDC panels, or refracted from three corresponding digital micromirror chips. The red, green and blue subbeams may be combined using dichroic mirrors, and the combined beam may be projected through a single lens onto screen SCR. Under ideal conditions, corresponding pixels in the modulated red, green and blue subbeams would be aligned in the combined beam. However, in actual practice, one may observe independent distortions and separation of the red, green and blue pixels in the combined beam because of various manufacturing imperfections. For example, the cells in each light processing chip may be independently distorted away from a nominal rectangular array. There may be slight variations in the alignment and pathlength calibration of the light processing chips with respect to the dichroic optics, etc. The distortions and separation of the red, green and blue pixels in the combined beam may be exacerbated as the combined beam passes through the lens and is projected onto screen SCR due to chromatic aberration (i.e. wavelength dependent refraction). Thus, the red, green and blue physical images on screen SCR are independently distorted away from an ideal rectangle, and corresponding pixels in the three physical images fail to overlap. Furthermore, the relative displacements between corresponding pixels in the three physical images are likely to vary in magnitude and direction across the screen.
According to a fourth model, a projection device may employ a single light processing chip (such as a digital micromirror chip) with a color modulation device (such as a color wheel) to generate a single output beam bearing all three colors in a time multiplexed fashion. The human visual system performs the color integration. The single output beam is projected onto the screen SCR through a single lens (or lensing system). The red, green and blue pixels in the single output beam are distorted due to chromatic aberration in the lens. However, because the source light beam is not split and recombined, many of the distortion mechanisms of the third model above are avoided.
According to a fifth model, a projection device may employ two light processing chips. A source light beam may be split into two subbeams. The first subbeam may be modulated with color information for a first color using the first light processing chip. The second subbeam may be modulated with the color information for the two remaining colors using the second light processing chip and a color wheel. The two modulated subbeams may be recombined using dichroic optics. The combined beam is then projected onto the screen SCR through a single lens (or lens system). The red, green and blue pixels in the combined beam experience chromatic aberration in passage through the lens. However, prior to interaction with the lens, the distortions and separation of the red, green and blue pixels in the combined beam may have magnitudes that are less than in the third (three-way split and recombine) model and larger than in the fourth (no split) model.
Thus, there exists a need for a system and method which could compensate for arbitrary chromatic distortions (i.e. distortions which separate the color components of a composite physical image) independent of the mechanisms by which such distortions arise within a projection device.
Computer system CS may rely on a graphics system to generate output video signals for projection devices P1 and P2 (and/or monitor device MD). Early graphics systems were only responsible for receiving data produced by the host processor and displaying it on monitor device MD. In essence, they acted as simple translators or interfaces. Modem graphics systems, however, incorporate graphics processors with a great deal of processing power. They now act more like coprocessors rather than simple translators. This change is due to the recent increase in both the complexity and amount of data being sent to monitor device MD and/or the projection devices. For example, modern display devices have many more pixels, greater color depth, and are able to display more complex images with higher refresh rates than earlier models. Similarly, the images displayed are now more complex and may involve advanced techniques such as anti-aliasing and texture mapping.
As a result, without considerable processing power in the graphics system, the host processor would spend a great deal of time performing graphics calculations. This could rob the host processor of the processing power needed for performing other tasks associated with program execution, and thereby dramatically reduce the overall performance of computer system CS. With a powerful graphics system, however, when the host processor is instructed to draw a box on the screen, the host processor may send a request to the graphics system stating “draw a box at these coordinates.” The graphics system may then compute the position and color of each pixel comprising the box. The host processor is thus liberated from the time consuming draw or render process.
Generally, a graphics system is a type of video adapter that contains its own processor to boost performance levels. These processors are specialized for computing graphical transformations, and thus, they tend to achieve better results than the general-purpose CPU used by computer system CS. In addition, they free up the computer's CPU to execute other commands while the graphics system is handling graphics computations. The popularity of graphical applications, and especially multimedia applications, has made high performance graphics systems a common feature of computer systems. Most computer manufacturers now bundle a high performance graphics system with their computer systems.
Since graphics systems typically perform only a limited set of functions, they may be customized and therefore are far more efficient at graphics operations than the computer's general-purpose CPU. While early graphics systems were limited to performing two-dimensional (2D) graphics, their functionality has increased to support three-dimensional (3D) wire-frame graphics, 3D solids, and now includes support for three-dimensional (3D) graphics with textures and special effects such as advanced shading, fogging, alpha-blending, and specular highlighting.
Prior art graphics systems use a conventional frame buffer for refreshing pixel/video data on a display device (e.g. monitor device MD, or projection device P1 or P2). The frame buffer stores rows and columns of pixels that exactly correspond to respective row and column locations on the display device. Prior art graphics system render 2D and/or 3D images or objects into the frame buffer in pixel form, and then read the pixels from the frame buffer during a screen refresh to refresh the display device. Thus, the frame buffer stores the output pixels that are provided to the display device. To reduce visual artifacts that may be created by refreshing the screen at the same time the frame buffer is being updated, frame buffers in most graphics systems are double-buffered.
To obtain more realistic images, some prior art graphics systems have gone further by generating more than one sample per pixel. As used herein, the term “sample” refers to information calculated at a particular point on an object or image, i.e. information such as one or more of color, depth (z), transparency, etc. For example a sample may comprise the following component values: a red value, a green value, a blue value, a z value, and an alpha value (e.g., representing the transparency of the sample). A sample may also comprise other information, e.g., a z-depth value, a blur value, an intensity value, brighter-than-bright information, and an indicator that the sample partially or completely comprises control information rather than color information (i.e., “sample control information”). By calculating more samples than pixels (i.e., super-sampling), a more detailed image is calculated than can be displayed on the display device. For example, a graphics system may calculate four samples for each pixel to be output to the display device. After the samples are calculated, they are then combined or filtered to form the pixels that are stored in the frame buffer and then conveyed to the display device. Using pixels formed in this manner may create a more realistic final image because overly abrupt changes in the image may be smoothed by the filtering process.
These prior art super-sampling systems typically generate a number of samples that are far greater than the number of pixel locations on the display. These prior art systems typically have rendering processors that calculate the samples and store them into a render buffer. Filtering hardware then reads the samples from the render buffer, filters the samples to create pixels, and then stores the pixels in a conventional frame buffer. The conventional frame buffer is typically double-buffered, with one side being used for refreshing the display device while the other side is updated by the filtering hardware. These systems, however, have generally suffered from limitations imposed by the conventional frame buffer, and by the added latency caused by the render buffer and filtering hardware. Therefore, an improved graphics system is desired which includes the benefits of pixel super-sampling while avoiding the drawbacks of the conventional frame buffer.
U.S. patent application Ser. No. 09/251,844 titled “Graphics System with a Variable Resolution Sample Buffer” discloses a computer graphics system that utilizes a super-sampled sample buffer and a sample-to-pixel calculation unit for refreshing a display device. The graphics processor generates a plurality of samples and stores them into a sample buffer. The graphics processor preferably generates and stores more than one sample for at least a subset of the pixel locations on the display device. Thus, the sample buffer is a super-sampled sample buffer which stores a number of samples that may be far greater than the number of pixel locations on the display device. The sample-to-pixel calculation unit is configured to read the samples from the super-sampled sample buffer and filter or convolve the samples into respective output pixels, where the output pixels are then provided to refresh the display device. The sample-to-pixel calculation unit selects one or more samples and filters them to generate an output pixel. The sample-to-pixel calculation unit may operate to obtain samples and generate output pixels which are provided directly to the display device with no frame buffer therebetween.
Thus, a system and method is desired for providing improved display capabilities including improved edge blending, hotspot correction, edge matching, distortion correction, and chromatic distortion compensation in a system having a super-sampled sample buffer.